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(54) Determining object motion from optical flow analysts 



(57) In a monitoring device for vehicles, an optical 
flow detection section determines an optical flow from 
an image taken with a camera. A background flow esti- 
mation section detennines a background flow, which is 
a flow for the camera image under the assumption that 



the image is a background, based on the motion of the 
vehicle. An approaching object detection section de- 
tects motion of an object in the surroundings of the ve- 
hicle by comparing the optical flow with the background 
flow. 
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Description 

BACKGROUND OF THE INVENTION 
[0001] 



oplical flows are widely known The s^^I^T^!' ! ° "^'"9 « P'^^^lity of cameras and a m^Hnn . ■ 

screen has a plurality of •^orizon.ai.y-dlvldrrrg^nsT^^^^^^^^ the vehil L A 

a magnrtude equal to or larger than a predeterr^ineJthreshold InH h ^" ^^^^^^d. which has 

approachrng object on an imago when such an apprrachij o iTe^ ' '° '^"^ motion of an 

an approaching object is recognized. ^PP'-oach.ng object ,s assumed. Based on the extracted optical ffow 

[00071 For monitorinn H,.H "H"caiTiow, 



30 



35 



40 



45 



SO 



55 



rOOOTI C«r.„7.„-. J "'"'"auiea optical flow, 

[0013] Japanese Patent Gazette No prrpi . 

o^^t using an optica, flow free «rom Snoe o/a^^^^^ « technique of detecting an approaching 

Vdl theoretically computed from a movement pZametTJV^^J, '^"^^ ^ ^'"^'^"^^ ^ ''^t^veen a motion 
.s computed according ,o the following expS^n us no vectoTT^ ^ """"^^ ^' '^^'^'^'^ f™'" 

vectordetectedfroman image. totheV.e.ect^aZro';:-^^ 

e i^ = ((Vdi - Vi).r1 \f + ((Vdi - Vi).»2i)2 
sMght .ho,d. •n.en.fore u.. 00^?!' 5 WPP«<I wllh Ih. mo„«ort„g a«*e loe«-. v.hi..,., ■ 

'«-«"ca "Rssrr^^^^ 

C00,6, *^=™S»'a,.ca™™po,„«,<,^..^,„^-,"^^B^,^ ^ 

o^M 15 laKen when the vehicle is running 
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along a curve. In the first related art, horizontally divided region L and region R are set as shown in FIG. 34A. Assuming 
that an approaching car exists in range AR1 in the region L as shown in FIG. 34B, the "direction of the nnotion of the 
assumed approaching car" is lower rightward as shown by the corresponding arrow in FIG. 34B. On the contrary, in 
range AR2, which also belongs to the range L but is different In vertical position from the range AR1 , the "direction of 
the motion of an assumed approaching vehicle" is lower leftward as shown by the corresponding arrow In FIG. 34B, 
which is completely different from the direction of the motion in the range AR1 . Thus, when the vehicle is running along 
a curve, the "direction of the motion of an assumed approaching vehicle" varies with the position of the approaching 
vehicle evein in the same region L. This makes difficult the detection of an approaching vehicle. 
[001 7] Even during running of a vehicle along a straight road, the magnitude of an optical flow is significantly different 
between the upper and lower parts of a screen. More specifically, in the upper part of the screen, which shows a region 
distant from the user's vehicle, a detected optical flow is very small. On the contrary, in the lower part of the screen, 
which shows a region very close to the user's vehicle, a detected optical flow is relatively very large. 
[0018] Therefore, if the upper and lower parts of the screen are subjected to processing with the same threshold, 
the accuracy of detection of an approaching vehicle is degraded with high possibility. For example, the threshold will 
be a very small value when determined based on the small flow in the upper part of the screen. If the lower part of the 
screen is subjected to processing with this threshold, noise tends to arise. On the contrary, the threshold will be a very 
large value when determined based on the large flow in the lower part of the screen. If the upper part of the screen Is 
subjected to processing with this threshold, most optical flows will be below the threshold, resulting In failure of detection 
of an approaching object. 

[0019] In the first related art, also, an approaching vehicle is detected using only optical flows having a magnitude 
equal to or more than a predetermined value. Therefore, a vehicle running alongside the user's vehicle at nearly an 
equal speed falls to be detected because the magnitude of the optical flow for this vehicle Is nearly zero. 
[0020] In the second related art, the turning vector generated by turning of the user's vehicle varies in magnitude 
and direction with the three-dimensional relative position from the camera of an object point related to the turning 
vector. Therefore, the tuming vector can be estimated only when the correspondence between a point on the camera 
image and a point on the three-dimensional real world is known. 

[0021] The above matter will be described with reference to FIG. 35. FIG. 35 shows the relationship between a 
camera image taken with a camera 2 and a three-dimensional coordinate system In the real world. The camera Image 
has Xi axis in the horizontal direction and Yi axis in the vertical direction. The real world coordinate system has Xw. 
Yw and Zw axes defined as shown in FIG. 35. That is. it is assumed that plane Xw-Zw Is a plane parallel to the road 
surface, direction Xw is rightward and leftward with respect to the user's vehicle, direction Yw is normal to the road 
surface, and direction Zw is forward and backward with respect to the vehicle. A camera coordinate system (Xc, Yc. 
Zc) is also set in which the focal point of the camera is the origin and the optical axis of the camera is Zc axis. Note 
that the directions of the axes are not limited to those specified above. These coordinate systems have relationships 
given by expression (1 ) representing perspective projection transfomaation and expression (2) representing coordinate 
transformation. 
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where f is the focal distance of the camera and r is constants determined by Internal parameters of the camera and by 
the position at which the camera is placed, that is, the position of the camera coordinate system in the real world 
coordinate system, which are known values. From the above relational expressions, it is found that a three-dimensional 
coordinate position in the real world corresponding to an arbitrary point on the camera image is on a straight line 
passing through the focal point of the camera. However, the position cannot be detennined uniquely without further 
infomnation provided. 

[0022] To state differently, as shown in FIG. 36, a position in the real world coordinate system can be transformed 
to a point on the camera image by use of the perspective projection transfonnation (expression (1)) and the coordinate 
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transformation (expression (2)). However these rplatinn^i ^ 

mation from a point on the cariera Image toTposi^n tthTrir'^fn'" '° ^^^"^^ ^«-«^e transfer- 

the turning vector must be detem,lned for respe'ctSXs on thl c^^^^^^^ 

LU023] lnthethirclrelatedart,asintheseconHraiot«^«^ p'^viueu. 

between the image position and the dStancTand ttfeS^^^^^^^ °' determine the correspondence 
£0024] In the fourth related art. a FOE exU ol duZT k! f 

running in the fi.t place. Therefore, .an optrclurifcre^^^;^^^^^^^^^^^^^^ "° «-ts during curlTe 

fO026] The Sixth related art. wS ^uTe t^a? m^^^^^^^ °" ^ ^^'^ ^^^'"9 - ^^^^ 

.etectlonwillheunsuccessfullnsuchacaserar^^^^^^^ 

SUMMARY OF THE INVENTION 

Ss^raro^i^Lit^^^^^^^^^^^ 

[0028] Specifically, the monitoring device of the nven'^^^^^^^^^^^ ^""""^'^'^ ^"^^ """'"^ ''""'^ '"""'"9 

w.lh a camera capturing the surroundings of a vehto e de " f ^ ^^""^ '"^^9^ taken 

.mage under the assumption that the Image ,s a bSkafouTbalH "^^'^9^°""'^ - an optical flow for the 

an Object existing in the surroundings of the vehiclfbv Tm.lrin 1 °' ^^'^''^'^^ motion o1 

[0029] According to the present invention tl^ btckarord flT- '"^ background flow, 

assumed as a background. Therefore, brcompaSna the hi^ ^" T^^' ^^^^^ « "-^era image is 

from the camera image, the motion of an otJecTexSino in '9^°""^,.^'°* ^'^ the optical flow actually deten^^ed 
.n addition, since the detection is perfom,ed b c^ari g h ~o3 '^^^ '''''''' ---'^^ 

the .mage, an approaching object can be deteJed prS even whfn T "P"^"' ""'^ ^^'^'^ P°int on 

a^:^^t?rr^^ 

SS'^rrgr^^^^^^^^^^^ 

The space model preferably includes m ieit a Z^^^ 

runn.ng. Othenvise. the space model preferably ?ncludi at least a ^aS <f, '"^''^ °" ^''^'^ ^'^^ ^ 
surface. ^ '"O'^fi' assuming a wall nornial to the road 

[0031] Alternatively, the monitorinq device of the or«.o„„i 

With a camera capturing the surroundingTof a veh^^^^^^^^ ""''T'"' ^P^'*^^' ^^^^ '-«9e taken 

m a real world coordinate system, based on the opSlS^^trof thTvph'^i """7 " °' ' °" » '-i 

the camera . capturing, and detects motion of a^n objecSrsrgrtrs^r ^^^^^^^^ « 



BRIEF DESCRIPTION OF THE DRAWINGS 
[0032] 



an Ackon,«„ model. ' '"""'"'^'■"^'""•.■>r»<»noeplual.lewaon,orami*g 
a vehicle during turning. ^ background flow, or a conceptual view illustrating the motion of 

no.^'s%%rcrrsr:g^^^ 

FIG. 8 is a flowchart showing he ope a ion o « 1 ''°'"P«"«°" «<^«fon. 

FIGS. 9A to 9D are conceptua ^iewT^Zstrr 

FIG. 1 0 is a conceptual view demTtrX h Jl o Ltim'l""^^ 

F:G^^Ji^:;l— ^rs 

.G..isanexampleofaca-ai— S^^^^^^^ 
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FIG. 14 Is a block diagram of a monitoring device of Embodiment 2 of the present invention. 

FIG. 15 is a view showing an example of a space model according to the present invention. 

FIGS. 16A and 16B are views showing the relationship between the distance L and background flows in the space 

model of FIG. 15. 

5 FIG. 17 is a view showing another example of a space model according to the present invention. 

FIGS. ISA and 18B are views showing the relationship between the width W and background flows in the space 
modelof FIG. 17. 

FIG. 1 9 is a flowchart showing a flow of processing of background flow estimation in Embodiment 2 of the present 

Invention. 

10 FIG. 20 is a view demonstrating how to estimate a background flow in Embodiment 2 of the present invention. 

FIG. 21 is another view demonstrating how to estimate a background flow in Embodiment 2 of the present invention. 
FIG. 22 is yet another view demonstrating how to estimate a background flow in Embodiment 2 of the present 
invention. 

FIG. 23 is yet another view demonstrating how to estimate a background flow in Embodiment 2 of the present 
15 invention. 

FIG. 24 is an example of an image taken with a camera mounted on a vehicle. 

FIGS. 25A and 25B are views showing the results of detection of approaching objects from the camera image of 
FIG. 24 according to the first related art. 

FIGS. 26A and 26B are views showing the results of deleclion of approaching objects from the camera image of 
20 FIG. 24 according to the present invention. 

FIG. 27 is a view of an example of a hardware configuration according to the present invention. 

FIG. 28 is a block diagram of a monitoring device of Embodiment 3 of the present invention. 

FIG. 29 is a conceptual view demonstrating a problem in hierarchical images. 

FIG. 30 is a block diagram of a monitoring device of Embodiment 4 of the present invention. 
25 FIG. 31 is a view demonstrating processing in Embodiment 4 of the present invention. 

FIG. 32 is another view demonstrating the processing in Embodiment 4 of the present invention. 

FIGS. 33A and 338 are views showing examples of placement of obstacle sensors according to the present in- 
vention. 

FIGS. 34A and 34B are views showing an example of a camera image of a scene in the rear from a vehicle, for 
30 demonstrating a problem of the first related art arising during curve running. 

FIG. 35 is a view illustrating the relationship between a camera image and a three-dimensional coordinates in the 
real worid. 

FIG. 36 is a view showing the relationship between a camera image and a real world coordinate system in the 
prior art. 

35 FIG. 37 is a view showing the relationship between a camera image and a real world coordinate system according 

to the present invention. 

FIG. 38 is an example of superimposition of optical flows for an approaching vehicle on a camera image during 
curve running. 

FIG. 39 shows background flows on the image of FIG. 38. 
40 FIG. 40 is an example of superimposition of flows determined by subtracting the background flows on the camera 

image. 

FIG. 41 is a conceptual view demonstrating how to detect the motion of the vehicle using a camera image. 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 

45 

[0033] Hereinafter, preferred embodiments of the present invention will be described with reference to the accom- 
panying drawings. 

Embodiment 1 

50 

[0034] In Embodiment 1 of the present invention, the surroundings of a vehicle are monitored in the following manner 
First, optical flows are determined using images from a camera capturing the surroundings of the vehicle. Secondly, 
the correspondence between respective points on a camera image and three-dimensional coordinates in the real worid 
is estimated as a "space model". As shown in FIG. 37, by use of this space nnodel together with the perspective 
55 projection transformation, a point on the camera Image can be precisely associated with real world three-dimensional 
coordinates. Thereafter, using the space model and estimated user's vehicle motion infomnation, optical flows under 
the assumption that points on the image constitute a background, not a moving object, are computed. The thus-com- 
puted optical flows are called "background flows". The background flows are compared with the optical flows actually 
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mS'^T^JifTTl'^T' '° "^'"'y ""^'^^ approaching object 

a backgroond flow and an optical flow at each po^S on the fml^^^^^ 1 k f '"^^^ comparison between 

and a distant approaching object, although the Optica t^^^ inning alongside the user's vehicle 

.^^ijreir^^"^^^^^^^^ 

IroSSo^^^^^^^^^^^^ 

putingan optical flow Vi from ■^n.agLtlk^nm^^e^^ ^VP"^^' '^'^'-^ -action 12 for com 

the motion of the vehicle; a spaL model esratirsSirn'14 'or L^^^^^^^ 

captunng; a background flow estimation section 15 "or el ir^a,J,a « ^T'"'^^^''^' °' « ^P^'^^ 'f'® camera 11 is 
user's vehicle and the space model; and an approicNna obl2, dl.?^'°""''/'^ ''''' "^"^^ '"°«on of the 

LrnLivel^a-m^r 

ne.ghbonng vehicle may be used, in combination with the camera mount^H 1'"^' ^ '^^^^^ °" « 

of such a camera Is effective in situations in which an aooZ^h!n J f "s^lf. The use 

such as at a crossroad with poor visibility approaching vehicle is hardly visible from the user's vehfcte 



75 



20 



^5 <Optical flow detection> 
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Kt^ro^^tTa^er^^^^^^^^ an apparent motion vector on an 

Widely known. One is a gradient method in whtej a co^rtSetaLnT ^ °' ^" °P«*=^' "^^^^ods are 

and the other is a block matching method using teS^matchSMin TT""^ differentials of an image is used 

I^erver--"-'— 

KheS^r^To^rcJ^^^^^^ requires a huge amount of processing 

(Gang Xu and Saburo Tsuji. 'Three-DimensioKsiorStsu ShuS t Tf ^'^^^^'^'^'^^''^ widely employed 
are produced by scaling down a given Image to a half oneSih on.^'^ 1 ^ ^Pacifically, hierarchical images 
hierarchical images, two points that are disLt from Lh otheNn an f^^^^^^^^ k°'' '""^'^ ^"'^ 'hese 
to each other in an image with low resolution (small sfze) In Jon^e^JtS^ ?'°'""°" ^'^^^^ ^•"J close 

onmages with low resolution. Thereafter, next tempiS n^iatcl^^^^^^^ the, template matching is first performed 
m the neighborhood of an optical flow obta nod f roZhSSStI 1^?' T P^^°^«d cn'V 

flow can be finally detem,ined for original images wfth hS rZSfon Sn ' ^ T^'"^ processing, an optical 
the processing time can be greatly reduced. ^ resolution. Since this method involves only local research. 

<Est|mation of motion of user's vehicle> 
r^Ls^e^ri™ 

This estimation method will be described with referencTto Flcf 2 and rNr^jT^^^ °' "'^ "^^^^ ^^'^'^l^- 

(two-wheel model) in which the state having no sidewav sHrifon Jr ^* ^ *°-calted Ackerman model 

[0041J FIG. 2 is a view showing an ^c^erZ^^7. ^■ I ^PP^'^f^ated is used, 

the vehicle tums around point olxLfng raTe^nZ ^ "° ^"^'"^ «^ -c"rs. 

turned^The turning radius Rs atthecenterbetweenlrrrrleeVaTi, ^'^^ ^'^^""9 ^^^^^ « 

angle p of the front wheels .3a and the wheel base 1 '^^'"^^^"^^^ ^ represented by expression (3) using a turning 



Rs= 



tanp (3j 



6 



): <EP, 



.1361543A2J_> 



EP 1 361 543 A2 



10 



15 



the rear wheels 3b moves from Ct to Ct+1 , the travel amount h is represented by expression (4) using the left and right 
wheel speeds VI and Vr. or the turning radius Rs at the center between the rear wheels 3b and the travel turning angle y. 

h=J^^dt=Rs.Y (4) 

[0043] From expressions (3) and (4), the travel turning angle y is represented by expression (5). 

nS 2 I . ^ 

[0044] Accordingly, the turning amount a of the vehicle with respect to the plane is represented by expression (6). 



[0045] The motion vector T from Ct to Ct+1 is represented by expression (7) when X axis is the vehicle travel direction 
and Y axis is the direction vertical to the vehicle travel. direction. 

20 1 

T = (Rs siny. Rs(1 - cosy)) = {^^(^'"'y'"' * (7) 

[0046] From expressions (6) and (7), the motion of the vehicle can be estimated if the left and right wheel speeds 
VI and Vr and the steering angle p are known. 
25 [0047] Naturally, the motion of the vehicle may also be detemnined directly with a car speed sensor and a yaw rate 
sensor, not only with the wheel speed and the steering angle infomialion. Otherwise, GPS and map infomnation may 
be used to determine the motion of the vehicle. 



<Space model estimation> 

30 

[0048] The space model estimation section 1 4 estimates a space model modeling the space the camera is capturing. 
The space model is used for determining the correspondence between a point on a camera image and real world three- 
dimensional coordinates, as described above. That is, from expressions (1) and (2). an arbitrary point on a camera 
image can be associated with a straight line in the real w!orld space passing through the focal point of the camera. The 
35 intersection point between the straight line in the real world space and the estimated space model is then detemnined, 
and in this way, the arbitrary point on the camera image can be projected in the real world three-dimensional coordinate 
system. 

[0049] Assume herein that the space model is prepared based on distance data for objects the camera is capturing. 
The distance data can be measured by use of binocular vision or a motion stereo method, for example, or by use of 
^0 an obstacle sensor using laser, ultrasonic, infrared, millimeter wave or the like if such an obstacle sensor is mounted 
on the vehicle. 

[0050] In the case of using a camera secured to an infrastructure such as a building, three-dimensional information 
on the space the camera is capturing is known because buildings and the like being captured by the camera very rarely 
change in shape. In this case, therefore, estimation of a space model is unnecessary, and a known space model may 

^5 be determined in advance for each camera. 

[0051] In the case of using a camera mounted on a vehicle, GPS and the like may be used to precisely locate the 
position of the vehicle. By locating the current position of the vehicle on detailed map information, a space model can 
be estimated. For example, if It is found that the vehicle Is running inside a tunnel from GPS and map data, a space 
model can be prepared from shape infonnation on the tunnel such as its height and length. Such shape information 

50 may be included in the map data in advance, or may be held on the side of the infrastructure such as the tunnel to be 
given to the vehicle via communication. For example, shape information of a tunnel may be transmitted to a vehicle 
that Is about to enter the tunnel by a communication means such as DSRC provided at the entrance of the tunnel or 
the like. This method can be applied, not only to tunnels, but also to roads, expressways, residential districts, parking 
lots and the like. 

55 
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cBackground flow estimation> 



a moving 

camera 2, which is moonl.d on iha vehkJe it^atlTI^^^S''^ ™"»" 

in ,h,a p,oiacion, ,ha peipeciva m^^. uZl:ri°T^X'S^^l:^^ " '^"1 
formation represented by expression (2) are used Wnt« thot th^ f , expression (1 ) and the coordinate trans- 

t estimated by the use 's vehicle motion estS," on 3''"^^^^ / ''^ ^ ^"^ ^° '""^ 

Changed based on the detemiined focal positioH the came a 2 fn £ ^a! w^rM h'°"'*^"'' (2) are 

this processing, the focal position of the camera in the3 wo ^^^0^ ^'^^ By repeating 

sistemiy precise position. ""^'^ '^"'"^ °""^'"ale system is kept updated, providing con- 

sifa^proirdrrto^iii:^^^^^ 

(NextXi, NextYi) represent t'he new Tuition o^^^e plT(^^^^^^^ 

the point (PreXi, PreYi) is assumed to be a ooint in th^ha^to ' 71 °" ''^""^'^ » when 

t. Thus, (NoxtXi-PreXi NextYiZY n?detZined a^a b J^ JT ' 

PreYi) on the camera image at timJ?.; is'ptrtTtS brCn.^sJsV'"" °" ^^^^ 

such as that the vehicle is running^traighrahfad and ^hant^Ta^i other s«uations 



<Approaching object detection> 
[0058] 



flow detection section 12 with the backaround f^w vh, nLf^^^ '"""^ ^""^ ^^'"^^^ '"^«9e by the optical 

detect approaching object candldate^or A no^r!^ detemiined by the background flow detection section 15. ,0 
candidate flows deterriined bX tw c^^^^^ po.io "^^^^^^ 'T'' "'''^^ '^'^ approaching ob ect 

[0059] FIG. 7 is a flowchart showing theXatton onJTf ow'.l °^ ^ ^" ^PP^^'^'^'^S °bject flow 
Of the optical flow Vi with the background llorid Ts oerfo^^^^^^ P^'^'^'P'^' comparison 

flow Vi is small in magnitude the S^Si^^x^^Z!^?^^ ^"^'^ difference. However, when the optical 

the recognition accu^^c^ ^o^ exa^^ t^e o^^^^^^ TTT. " "^""'"^ "^^'"♦^"-9 

an equal speed is very small in magnftude SLo triSc^o'J^^^^^^^^^^^^ T^T "'^'^ ^"'"^'^ ^ 
timing at which the image is taken For examole he onti^I?« J ^ °^ ''^^"S®^ ^^^^ '"^^ance with the 

pointing to the opposite'direction n otherrovercort^^^^ users vehicle in some instances while 

smaller In magnitude than a predetermined va^e The anniP ^ embodiment, when the optical flow Vi is 

(S43) is used to Improve the Jellabil^Jme reeogin ^ " comparison crtterion 

direction infomiation. the comparison thereof wjS!,e bafirotl!fnrf « sufficiently reliable in its 

(S42). In the comparison, if the absolu e Jtt angrdiSer^^^^^^ " ""'"^ ''^^ 

VdiisequaltoormorethanapredeterminedvareTH r^^^^^^ 

flow Vdi. and thus recognizeS to be How oTl^^'p^J;-^^^^^^ considered different from the background 

sufficiently small (NO in S42). the optical flow Jhs cL J m Z h u ^ V ^"^"'"^^ °^ ^^"^ ^"^'^ difference is 
flow Of an approaching object (S45V The rreshold TH if n2 ^"'^ -"ecognlzed not to be a 

preferab^ about n^. L Ll ^l^y T. Zl::^Z]n1^^^^ l^JZlT" T T'°'' ""^^ 

for comparison in the case of YES in step S41 is to *.iimin=f. tiT ? . 5 ^ '"'o'''"ation is used 

users vehicle may be recogni7ed an appmacWno or.S 1^ ^^T ,'^"^ " "^"^'"^ ««'«y f™"* the 

be used as a recognition crUerion. ^PP^^'^^'^S °''J«'=« Naturally, the absolute of the flow vector difference may 
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[0061] If the optical flow Vi is not sufficiently large (NO in S41), the angle difference is not usable for comparison 
with the background flow. In this case, the magnitude of the flow is noted. That is, whether or not the absolute of the 
vector difference between the optical flow Vi and the background flow Vdi is equal to or more than a predetermined 
value THvdi is determined (S43). If it is equal to or more than the predetermined value THvdh ^^e optteal flow is recog- 
5 nized to be a flow of an approaching object (S44). Otherwise, it is recognized not to be a flow of an approaching object 
(S45). As the situation in which the optical flow Vi is small in magnitude and represents an approaching object, the 
following two cases are considered. 

1) An approaching object is running alongside the user's vehicle at nearly an equal speed. 
10 2) An approaching object is running in the distance. In either case, since the background flow Vdi can be precisely 

determined, highly accurate recognition of an approaching object can be realized. The threshold THv^j Is preferably 
about 0.1 [pixel]. 

[0062] In step S43, since the magnitude of the optical flow Vi is sufficiently small, only the magnitude of the back- 
us ground flow Vdi may be compared with a predetermined value, not the absolute of the vector difference between the 
optical flow Vi and the background flow Vdi. In this case, the optical flow may be recognized to be a flow of an ap- 
proaching object if the background flow Vdi Is sufficiently large, and recognized not to be a flow of an approaching 
object if it is sufficiently small. 

[0063] The thresholds THyj, TH^rg and THv^i used (or the flow comparison described above may be set as the 
20 function of the position on an image. For example, the thresholds THyj and THvdi n^iay be set small for a position at 
which the background flow Vdi is small, and may be set large for a position at which the background flow Vdi is large. 
This enables precise recognition of an object in the distance while allowing suppression of noise influence. 
[0064] FIG. 8 is a flowchart showing the operation of the noise reduction portion 1 6b. Since noise is included in those 
detected as approaching object flows by the flow comparison portion 16a, the detection accuracy will be degraded if 
25 all of them are accepted as approaching objects. The noise reduction portion 16b prepares a model of noise and an 
approaching object and compares the model with approaching object flows (approaching object candidate flows) de- 
tected by the flow comparison portion 16a, to detect only an approaching object. 

[0065] First, if an approaching object candidate flow is noise, approaching object candidate flows similar to this flow 
will not be detected in succession temporally or spatially. On the contrary, if an approaching object candidate flow is 

30 not noise but related to an actual approaching object, which has some degree of dimensions, similar approaching 
object candidate flows should occupy an area corresponding to the dimensions of the approaching object spatially. 
[0066] In view of the above, neighboring approaching object candidate flows are associated with each other, and 
portions related to the associated approaching object candidate flows are combined together, to provide a combined 
approaching object candidate region Ai (S51). The area Si of the approaching object candidate region Ai is obtained 

35 (S52), and is compared with a predetemnined value THgi (S53). At this time, the predetermined value THsj is set small 
when the approaching object candidate region Ai is apart from the camera, and set large when it is near the camera. 
If the area Si is less than the predetemnined value THgi (NO in S53), the region Ai is detennined as noise (S54). If the 
area Si Is equal to or more than the predetermined value THsi, the process proceeds to step S55. 
[0067] In step S56, noise reduction is perfomned by modeling the approaching object. Since the camera is mounted 

40 on the vehicle, approaching objects should be cars, motorcycles, bicycles and the like, which are all objects running 
on a road. In view of this, whether or not the approaching object candidate region Ai exists on a road surface in a space 
model is detemnined (S55). If not, the region Ai is detennined as noise (S54), and the process proceeds to processing 
of the next frame. If at least part of the approaching object candidate region Ai exists on a road surface, the process 
proceeds to step S56. 

45 [0068] In step S56, filtering in the time direction Is performed. An approaching object will never appear or disappear 
abruptly on the screen. Therefore, the region of an approaching object should exist on several continuous frames. 
FIGS. 9A to 9D show a situation in which a vehicle is approaching to the user's vehicle from the rear during straight- 
ahead running. At time t, an approaching car 6 on an image is as shown In the left view of FIG. 9 A, for which an 
approaching object candidate region At is detected as shown in the right view of FIG. 9A. At times t-1 , t-2, t-N a 

so little before the current time t, approaching object candidate regionis A^.^, At.2, .., A^.^j are detected as shown in FIGS, 
9B, 9C and 9D, respectively. Assuming that the time intervals are sufficiently small, the approaching car 6 does not 
move so largely on the image, allowing the respective regions Aj.^, A^.g, Aj.^, to partly overlap the region At. If an 
approaching object candidate region has been generated due to vibration of the camera and the like, the candidate 
region should be detected only for a short time, 

55 [0069] In view of the above, the percentage at which an approaching object region candidate exists in a region of a 
frame corresponding to the approaching object candidate region Ai is compared with a predetermined value for pre- 
ceding several frames. (S56). If the percentage is lower than the predetermined value, the possibility that the approach- 
ing object candidate region Ai is an approaching object Is low. Therefore, this approaching object candidate region Ai 
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IS left behind (S57), and the process proceeds to processing of the next frame. If the percentage is higher than the 
predetermined va^.e ,n preceding several frames, the approaching object candidate region Ai is detim ne^rlted to 
an approachmg object, and the process proceeds to processing of the next frame (S58) For exampte rSoraa^^^^^^^ 
rnrlfniT,''"'^'^ determined as an approaching object when six or more frames oS oTten a7e aSTablL 

[0070] If rt ,s assumed that the camera is mounted to point to the front or rear from the vehSle and Z ^prT^.c, 
Objects are only passenger cars, images of approaching objects taken wrth the camera are only those ofpassenq^^^ 
cars viewed from front or rear. In this case, the size of the approaching object region can be limited to about 2 m fn 
width and about 1 .5 m in height from the road surtace. In view of this. 'the size o?the?pp^altroSercand^^^^^ 
ZZT7 '' V"'' '"^ °' '"^^ approaching object candidate regb^may L trs""^^^^^^^ 

innrS!!^/ approaching object flows existing within the region Is maximum. By setting in this way, whetheran apprJach 
mg object or no.se can be detemiined depending on whether or not the number of approaching obje^ lows inTded 
;"ac: o'fTpfss? Sa'^^"^'^^^^ '-9- ^ predetermined value. This' processing nla?.;e:;eS:d in 

S^IIL,!^ described above, in this embodiment, the motion of an object existing in the surroundings of a vehicle can 

ro072r Sen w^^^^^^^^ "''"'"^ ""'"""'"^'^ '^"^ « '^^'"^^^ "^^^e with f backgrouSi f low 

tJeStecMonroer?^^^^^ " ^ approaching object can be detected precisely because 

Ik!, f ? r L companng an optical flow with a background flow at each point on an image As for an 

Object o Which an optical flow determined from an image is small In magnitude, such as a vehlcirrZng abngslde 

Lo!nfotth ''T"' «PP™«-^'"9 oP"cal flow is greatly different from the backoZn^^fow a, t Je 

point on the image. Therefore, such an object can be easily detected. 'varouna now aime 

<Other examples of background flow estimation> 
(Example 1) 

ESfIni /''f "P^*'^^' "7 «^«t«rmi"ed in the case shown in FIG. 5, in which the camera 2 moves at the turning radius 

?. hTh r 7 V ''^ ' '^""^^^^ 2 '=«P'"""9 «t a", is equal to thertfcatf low 

to be determined when the camera 2 does not move but the object 5 the camera 2 is cap u ring mo^eri tJe turj ina 

real wo'ri'd coo°rHinrt t'"" ^'^^'^r'^' '"^^ °P"-' ^'^ <^«'--ined when the camera' movL by TveLor J i the 
L vl^or V " ^ '° Objects the camera is capturing move by 

[00741 In view of the above, the background flow may be determined, not based on the motion of the camera but 
b^Tgrrffoi^^'alXZ r i~ - - showmg^^g^S 

t«°,I!! ^'IT:^^ '"^^^ ^" ^'^'"''"^ P°'"* (P^'^'- P^^Yi) on a camera image taken at time t-1 is projected 

1h.1T .''^^^-'^''"^"^^"f ' coordinates (PreXw. PreYw, PreZw) using the space model esSed by thL space 

sion (1 ) and the coordinate transfomiation represented by expression (2) are used 

[0076] Thereafter, the position of the real world three-dimensional coordinates (PreXw PreYw PreZwl d^^t^rminoH 

umZi rir'" "r ?r ' '^'^'^ '° ^^^'^'^ ^^^^^ - «-°-t ^ ofThe ve^^^fe tzz:t 

1 until time t. In other words, the position of the real world three-dimensional coordinates (PreXw PreYw PmZw^s 
rotated around the rotation center coordinates O at the rotational angle y corresponding o the7ravel amounf h to 

NextYw NextZw) detemi.ned ,n step S22 is then projected back to a point (NexlXi. NextYi) on the cam^a I^aqeS 

n -^" ^' ('^^^'^'-P^-X'. NextYl-PreYI) Is detennined as a baigrrnZw (S24) 

[0077] It 'S possible to regard the coordinates (NextXw. NextYw, NextZw) determined in step S22 as a space model 
at time t predicted from the space model at time t-1 . Therefore, by continuing the processing described abovri^^^^ 
space model may be updated. This updating of the space model may be perfo^ed only for the poXn^detet^nt^ as 

(Example 2) 

El '"T^ T described above, a point on a camera image was transfomied to real world three-dimen- 

sional coordinates, and a space model assumed in the real world three-dimensional coordinate system was us^ for 

FIGTaTa'Zlrt sfowT'r """" ^"^^^^'"^ '"^^ P^^°--^ camera cordite sySer^ 

thfcnfo! mlnTM ^ ^ processing of background flow estimation in this method. According to this method 
the space model must be expressed in the camera coordinate system. Since the camera coordinate system isTone 
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to-one correspondence with the real world coordinate system, the space model assumed in the real world coordinate 
system can be easily transfomied to the camera coordinate system according to expression (2). 
[0079] First, an arbitrary point (PreXi, PreVi) on a camera image taken at time t-1 is projected to a three-dimensional, 
position (PreXc, PreYc, PreZc) in the camera coordinate system using a space model expressed in the camera coor- 

5 dinate system (S31 ). In this projection, the perspective projection transformation represented by expression (1 ) is used. 
[0080] Thereafter, the camera coordinates (PreXc, PreYc. PreZc) determined in step S31 is relatively shifted with 
respect to the vehicle 1 based on the travel amount h of the vehicle 1 from time t-1 until time t. In other words, the 
position of the camera coordinates (PreXc/ PreYc, PreZc) is rotated around the rotation center coordinates O at a 
rotational angle y c the camera coordinate system corresponding to the travel amount h, to determine camera coor- 

10 dinates (NextXc, NextYc, NextZc) (832). The position of the camera coordinates (NextXc, NextYc, NextZc) determined 
in step S32 is then projected back to a point (NextXi, NextYi) on the camera image (S33). Thereafter, as in step S15 
in FIG. 4, (NextXi-PreXi, NextYi-PreYi) is detemiined as a background flow (S34). 

[0081] As described above, the space model is used for transformation from a camera image to the real vyorld co- 
ordinate system or from a camera image to the camera coordinate system. It is therefore possible to express the space 
15 model in the form of a transformation expression from a camera image to the real world coordinate system or a trans- 
formation expression from a camera image to the camera coordinate system, 

[0082] FIG. 13 is an example of a camera Image with background flows according to the present invention indicated 
as arrows. As described before, a background flow is used for comparison with an optical flow actually determined 
from a camera image and serves as a detection criterion for an approaching object. As is found from FIG. 13, the 
20 background flows reflect the curve of the road. In other words, it is directly understood from FIG. 13 that the method 
according to the present invention can detect an approaching object accurately even on a curve, compared with the 
prior art. 

[0083] According to the present invention, only an object approaching the user's vehicle can be detected precisely 
Using the detection results, an image emphasizing only an approaching car may be displayed, or existence of an 
25 approaching car may be warned with sound, image display, vibration of the steering wheel and/or the seat, lighting of 
a hazard lamp such as a LED or the like, for example. Further, In a hazardous situation, the steering wheel and/or the 
brake may be switched to automatic control to evade contact or collision with an approaching car. 
[0084] Hereinafter, emphasizing display using background flows will be described. 

[0085] FIG. 38 shows an example of an Image in the rear from the user's car running along a curve, in which only 

30 optical flows corresponding to an approaching vehicle are superimposed. In FIG. 38, vehicles A and B apart from the 
user's vehicle by nearly an equal distance are approaching the user's vehicle at nearly an equal speed. As shown in 
FIG. 38, the user can recognize the approach of the vehicle A running on the outer lane of the curve by optical flows 
(arrows) superimposed. However, the user cannot recognize the approach of the vehicle B running on the inner lane 
of the curve because the optical flow for this vehicle is substantially zero. It is therefore found that during curve running, 

35 mere superimposition of optical flows may sometimes fail in recognizing existence of an approaching object. 

[0086] The reason for the above is that background flows generated by running of the user's vehicle along a curve 
and flows generated by the motion of the approaching vehicle cancel each other. FIG. 39 shows the background flows 
in the case of FIG. 38. In FIG. 39. arrows A2 and B2 represent background flows corresponding to the positions of the 
vehicles A and B, respectively. As is found from FIG. 39, the background flow 82 at the position of the vehicle B is 

40 opposite in direction to the motion of the vehicle B. 

[0087] To make only the motion of an approaching vehicle conspicuous, the background flows are subtracted in 
terms of vector from the determined optical flows, and the resultant flows are superimposed on the camera image. 
FIG, 40 is an example of the resultant display. Unlike FIG. 38, flows are displayed for both the vehicles A and B, so 
that the user can recognize existence of the approaching vehicles without fail. 

45 [0088] Naturally, It is also possible to combine approaching object flows together as described above to determine 
an approaching object region, and emphasize the approaching object region by framing the approaching object region 
as shown in FIG, 266 to be described later. Further, as the approaching object flow or the approachirig object region 
is larger in magnitude, the approaching vehicle is considered closer and thus the risk factor is considered higher. 
Therefore, the color line thickness., line type and the like of the frame may be switched with the magnitude of the 

50 approaching object flow and the approaching object region. 

[0089] In place of an image or together with an image, sound may be used for warning. In this case, it is effective if 
sound is heard from the direction in which an approaching object exists. The magnitude, melody, tempo, frequency 
and the like of the sound may be changed with the risk factor. For example, when a car is gradually approaching from 
the rear on the right, small sound may be generated from behind the user on the right. When a car is rapidly approaching 

55 from the rear on the left, large sound may be generated from behind the user on the left. 

[0090] In this embodiment described above, the motion of the user's vehicle was estimated from the rotational speed 
of the wheels and the steering angle of the steering wheel. Alternatively, an image taken with the camera mounted on 
the vehicle may be used to detect the motion of the vehicle (camera itself). This technique will be described. 
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the road surface on an image. Vhe geZZ ^Tnox leZ^^^^^^ ^"^'"^'"^ ^''^^'^'^ ♦''^ "^""o" of a point on 
at high speed, the motion oT the camera 2 L 21^^^^^ "^^"'^ the vehicle is running 

[0092] FIG. 41 is a diagrammatic view demonSng th^S Ze des^^^^^^^^ ''V'T '"""^ ^=^^'"""9 
that the coordinate value at a point P on a rest plane in thJ^ZlT ''^^^"''^'^ ^"^^^ ^s shown in FIG. 41 , assume 
PE- (x-, y. zr The motion of a camera 2 is reSestted bv ro^^^^^^^^ ''^"^es from PE = (x, y, z) to 

the motion of the camera 2 is expressed as fo tows ""'^ ^"'^ <^''' "^V- ^^J- 
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100931 Assuming that the rest plane is expressed by z - ax + b« * ^ th« f • 

the Shift Of the point P. (u, v) -> (u', V) in the ur^age coordinaUo^ sjslem ^"""^ ^'^P^e-^ions are established for 
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C c" C 

[00941 Assuming that the corresponding points (u vl and (u- v\ in tho ... 
40 a, b and c of the expression representing the rest plane are known he ^b^^' '^''""^ ""'^ coefficients 

of the unknown parameters (wx. wy. wz tx, ty tz) 1 foSS^s ' expressions can be reorganized in temis 
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[0095] By expressing the above as AC = R, the term of the rotation and the translation representing the motion of 
the vehicle can be computed using the method of least squares. 
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Embodiment 2 

[0096] FIG. 14 is a block diagram conceplually showing the basic configuration of a monitoring device for vehicles 
of Embodiment 2 of the present invention. In FIG. 14. the same components as those in FIG. 1 are denoted by the 
same reference numerals, and the description thereof is omitted here. This embodiment is different from Embodiment 
1 in that a space model estimation section 14A estimates a comparatively simple space model using user's vehicle 
motion information estimated by the user's vehicle motion estimation section 13. 

[0097] In general, various objects such as buildings, utility poles, signboards and trees exist on the road surface in 
the circumstances in which a vehicle runs. The camera 1 1 therefore captures various objects such as the road surface, 
buildings, utility poles, signboards, trees and the sky. Herein, a method for presuming various objects captured by the 
camera 11 using a simple space model will be described. 

[0098] Among various objects captured by the camera, the "road surface on which the vehicle has run" will be cap- 
tured almost without fail when the camera is placed to point to the rear downward. Therefore, a "road surface model" 
modeling the road surface on which the vehicle has run is first used as a space model. The road surface model, which 
includes infinite extension of the road surface, can be precisely estimated as long as there is no abrupt change in 
gradient. Even for a road changing in gradient, like a slope and a severe up and down road, a road surface model 
considering the influence of the gradient of the road surface can be prepared using a sensor such as a gyro sensor. 
[0099] However, such a road surface model is not necessarily usable for the entire region of the camera image. The 
reason is as follows. It is considered that the camera is nonnally placed in a nearly horizontal state. In this case, the 
road surface does not exist in the portion of the camera image higher than the horizon. Naturally, the camera can be 
placed so that the road surface occupies the entire region of the camera image. However, this placement narrows the 
coverage allowing monitoring with the camera and thus is not practical. 

[0100] In this embodiment, a wall normal to the road surface is presumed as a space model in addition to the road 
surface model. This space model is hereinafter called a "wall model". 

[0101] FIG 15 is a view showing an example of the wall model. Referring to FIG. 15, range VA defined by two straight 
lines extending from a camera 2 mounted on a vehicle 1 is the coverage the camera 2 can capture. A wall model MW 
shown in FIG 15 is the simplest wall model, in which a wall normal to the road surface is presumed at a position in 
the rear from the vehicle 1 by a distance L. The wall model MW is presumed to be large enough to cover all the region 
of the coverage VA of the camera 2 that are not covered with the road surface model MS. The region of the coverage 
VA of the camera 2 expanding between the camera 2 and the wall model M W constitutes the region of the road surface 
model MS. Therefore, when the vehicle 1 is running straight ahead, the camera image taken with the camera 2 is 
composed of the wall model MW in the upper part and the road surface model MS in the lower part. 
[0102] In use of a wall model as shown in FIG. 15, detemriination of the distance L to the wall model MW raises a 
problem. FIGS. 16A and 16B are views showing examples of background flows on a rightward curve, in which an-ows 
indicate background flows determined and white lines indicate white lines on the road surface. MWA denotes a region 
in which background flows were determined using the wall model MS. and MSA denotes a region in which background 
flows were determined using the road surface model MS. FIG. 16A shows the results obtained when the distance L is 
sufficiently long and FIG. 16B shows the results obtained when the distance L is sufficiently short. By companng FIGS. 
16A and 16B with each other, how the background flows change with the magnitude of the distance L Is observed. 
[0103] That is, it is found that the magnitude of the background flows in the wall model region MWA is greatly different 
between FIGS. 16A and 168. However, as described before, the condition that "part of an approaching object is in 
contact with the road surface" has been satisfied in the processing of approaching object detection. Therefore, back- 
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height. The region of the coverage VA of the camera 2 ^xi^tin« h^L 1 ?T'"^ ^ i"fi""e 

constitutes the region of the road surface model MS '^'^ "^^1 and IVIW2 
camera image talcen with the camera JrcoTposed of t^eCand rinM I ^''J'" ' ^"""'"^ 
and the road surface model MS in the lowe7part ,n F G the vehide ' ''''^ """^ "^^^^ P«« 

ir Lm^rcfmri^raT^^^^^ - - - °' transformation expres- 

[01 07] First, the road surface model will be deschh.H Tco ^^"'"^'"^^^ system will be described in detail. 

isafiat surface having no gradient fhirdtJrlld^^r^^^^^^^^^ 
coordinate system. Therefore, by subsmuling the re^ln::^:Z^^^^^^^ 
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center position between the rear wheels oHhe vehicZ aUimJ ^.n^Ih f 1'^'"*'"""''°"^' ^^"^'^'"^'^ ^y^'^"^ 
that the turning radius R can be positive or negare Ln^ sLS^n '^^"'^^ °' °)- Assume also 

Wise and negative when the vehicle 1 turns cSkwise P°^«"'« '^e vehicle 1 turns counterclock- 
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(Xw - R)2 + 2w^ = (R ± w/2)^ 
[0111] Using this expression together with exorpcsQinnc h\ /o. 
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[0112] The position Pw (Xw, Yw, Zw) can be determined uniquely by adding further conditions that 

1) an object appearing on the camera image does not exisl behind Ihe camera 2 (Ze > 0) and 
20 2) the space model is at a position higher than the road surface (Yw ^ 0). Expression (9) serves as the transfor- 

mation expression for the wall model from the image coordinate system to the real world three-dimensional coor- 
dinate system. 

[0113] In use of a wall model as shown in FIG. 17, determination of the distance W between the walls MW1 and 
25 MW2 raises a problem. FIGS. 18A and 18B are views showing examples of background flows on a rightward curve, 
in which arrows indicate background flows determined and white lines indicate white lines on the road surface. MW1 A 
denotes a region in which background flows were determined using the wall model MW1 on the left side as is viewed 
from the camera 2, MW2A denotes a region in which background flows were determined using the wall model MW2 
on the right side as is viewed from the camera 2, and MSA denotes a region in which background flows were determined 
30 using the road surface model MS. FIG. 18A shows the results obtained when W is sufficiently small, and FIG. 18B 
shows the results obtained when W is sufficiently large. By comparing FIGS. 18A and 18B with each other, how the 
background flows change with the magnitude of the distance W is observed. 

[0114] That is, it is found that the background flows in region BA1 neighboring the boundary between the left and 
right wall models are greatly different between FIGS. 18A and 188. However, as described before, the condition that 
35 "part of an approaching object is in contact with the road surface" has been satisfied in the processiiig of approaching 
object detection. Therefore, an error of background flows in the wall model regions MW1 A and MW2A. if any, will not 
cause a large problem as long as background flows in the road surface model region MSA have been determined 
precisely. 

[0115] That is, in wall models as shown in FIG. 17, unlike the model shown in FIG. 15 in which a wall is assumed in 
40 the rear from the camera, background flows in the boundary region between the wall models and the road surface 
model do not vary so greatly with the magnitude of the inter-wall distance W. This enables accurate detection of an 
approaching object. For example, the inter-wall distance W may be set at about 10 m. 

[0116] The inter-wall distance W can be measured with an obstacle sensor using laser, ultrasonic, infrared, millimeter 
wave or the like, or by binocular vision or a motion stereo method. Alternatively, texture information of the road surface 

45 may be used to extract a region other than the road surface from the camera Image, and the extracted region may be 
assumed as a wall. Othenwise, only part of the space model may be measured with a range sensor, and the remaining 
part may be modeled using a plane or a curved surface as described above. The inler-wall distance W may be deter- 
mined using the number of lanes of the road on which the vehicle is currently running determined from a camera image 
or via communication. For determination of the number of lanes from a camera image, white lines may be detected 

so and the number of lanes may be detemnined from the detected number of white lanes, for example. 

[01 17] The shape of the space model may be switched based on GPS information and map data as in a car navigation 
system, time data from a clock and the like, operation data of wipers, headlights and the like of the vehicle and the 
like. For example, if it is found from GPS information and map data that the vehicle is currently running in a tunnel, a 
space model including a ceiling may be determined. Altematively. information that the vehicle is currently running in a 

55 tunnel may be obtained via a communication means such as DSRC installed at the entrance of the tunnel. Otherwise, 
since the headlights are generally lit during running in a tunnel, it is determined, in association with time data, wiper 
data and headlight data, that the veh icie is running in a tunnel when the headlights are lit although it is neither nighttime 
(estimated from the time data) nor does it rain (estimated from the wiper data). Based on this detemnination, the space 
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-» detemtlnec a. the -cM .uZTn^LSl, "ana ^te hl™,lt^ = P""' -='"*9 the t,all TR Is 

?:s=r.T.ra"?,„^irr.::i:,^r^^^^^^^ 

vehicle 1 is running in an estimated space modeMh^ ToTnt 1^ ^^'^ assuming that the 

the straight line LN and the spa^f mode, (MW2 In RG S^^^^^ P^"* 

" Sxiirrp^rro^^th^^^^^^^^ 

the new position (S66). It is then co^tdt wS^^^^ expression (2)) are updated to conform to 

camera image at the new camtra ^^^567^ ^JiH™ f [ r""'"* coordinate point PreRi is located on the 
image is reaSzed using expJessfon (ifandSaLd elrlTn^Sr """^ — « 

rUro^trcrS'lLTattr^^^^^^^ descr.eda.ove^nd.ates the pos.on o, the 

time t, that is, the point PreRi is a po nTon a SacSound o£^^^^ Thl P°'nt PreR. has not moved from timet-1 until 
the point PreCi on the camera ImagTlt tir^e ^?S Zj^^^^^ ITrt TT J^'*^™""^ "^'^ ^FL detem,ined when 
PreXi NextYI-PreVn ^c?fifi^ r„ JJj^^ Z ^^"'^^^ as part of a background can be expressed as (NextXi- 

shown rn RG. ^3 iVbe iS'ned ""'^'^^'"^ ^" ^^'"^^ °" ^ « --9-- 'background io:: aL 

2?A 25VSnd 26B pTifr " embodiment will be described with reference to RGS 

isab^'unfmaSt^Sttum.!^^^^^^ 

however fails to be detected as an approaSa obiS b.ll! if k passenger car VC1 in the range AR1 

indicated by the arrow On the conLT he plsLnter^^^^^^^ °S ""'"""^ "^'^'^^''^ 

preaching object. The reason is that siL he use" ve^S ,^ r^nnino a S« " ""'"'^"'^ 
as indicated by the arrow is given to the passenaer car vSp «r L ^ .u""'^' ^" ^P"^^' 
above, in which the range dSected as ara^rofl!=htn f ^ ^^^f ♦^^^ t»ie processing described 

approaching object falls'duJi^gtu^eTnnlna ' " " 

[0127J this embodiment, background flows in the ranaes AR1 anri A ar/» 

in FIG. 26A. That is, in the range AR1 includino the nZ!no«r T.r x/Jf determined as shown by the arrows 

flow points lower rightward as shown n RG MA thrh^^^ . « ^" «PP''oaching object, while the optical 

-..tcand ,10. alo poLrerXa?™^ T.-^^^^^^ 
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range AR2 is recognized as a background, not an approaching object. FIG. 26B shows the results of the processing 
described above, in which the range detected as an approaching object Is framed. In this way, according to the present 
invention, unlike the prior art, an approaching object can be detected without fail even during curve running. 
[0128] A monitoring device for vehicles using a road surface model is described in Japanese Laid-Open Patent 
5 Publications No. 2000-74645 and No. 2001 -2661 60. However, the techniques described in these publications neither 
use the road surface model for determination of background flows, nor have an object of detecting an approaching 
object during curve running. These disclosures are therefore different in the problem to be overcome from the present 
invention. 

[01 291 To state specifically, in the former publication, optical flows generated from vehicles existing within a monitoring 
10 range are first detected, and using the detected optical flows, the correlation between the user's vehicle and the neigh- 
boring vehicles is monitored. A feature of this technique is limiting the range for detection of optical flows to shorten 
the processing time, and for this purpose, a road surface model is used. In other words, the space model is not used 
■ for detection of an approaching object flow unlike the present invention. Actually, this technique employs a method 
using virtual FOE disclosed in the aforementioned fifth related art for detection of an approaching object flow during 
15 curve running, and thus has the same problem as that arising in the fifth related art. That is, precise detection of an 
approaching object is unavailable on a curve having no white line. 

[0130] The latter publication uses three-dimensional motion of each point on a screen. Specifically, for each point 
on a screen, an optical flow as two-dimensional motion on the screen is first determined. Based on the determined 
optical flow and vehicle motion information, three-dimensional motion of each point in the real world is computed. The 
20 three-dimensional motion is traced with time, td thereby^estimate a space model for the space in which the vehicle is 
actually running. In the thus-estimated space model, an object different in motion from the road surface is detected as 
an obstacle. This technique, in which the motion of each point is determined completely three-dimensionally, has a 
problem that computation cost Is very high, and thus finds difficulty in implementation. 

25 <Example of hardware configuration> 

[0131] FIG. 27 is a view showing an example of hardware configuration for implementing the present invention. 
Referring to FIG. 27, in an image processor 20. an image taken with a camera 2 mounted in the rear of a vehicle, for 
example/is converted to a digital signal by an image input section 21 and stored in a frame memory 22. A DSP 23 
30 detects an optical flow from the digitized image signal stored in the frame memory 22. The detected optical flow is sent 
to a microcomputer 30 via a bus 43. A vehicle speed sensor 41 measures the travel speed of the vehicle, and a steering 
angle sensor 42 measures the steering angle of the vehicle. Signals representing the measurement results are sent 
to the microcomputer 30 via the bus 43. 

[0132] The microcomputer 30 includes a CPU 31 , a ROM 32 storing a predetermined program for control, and a 
35 HAM 33 storing computation results from the CPU 31 , and determines whether or not an approaching object exists in 
an image taken with the camera 2. 

[0133] Specifically, the CPU 31 first estimates the motion of the vehicle from the travel speed signal and the steering 
angle signal sent from the vehicle speed sensor 41 and the steering angle sensor 42. The CPU 32 then estimates a 
trail of the running of the vehicle based on the estimated motion of the vehicle. That is, the CPU 32 connects the 
40 estimated motion of the vehicle with past trail information stored in the RAM 33, to obtain a trail until the cun^ent time. 
This new trail information is stored in the RAM 33. 

[0134] The CPU 31 then estimates a space model using the trail information stored in the RAM 33, and detemiines 
a background flow. The determined background flow is compared with an optical flow supplied from the image processor 
20, to detect an approaching object flow and thereby detect an approaching object. 



45 



Embodiment 3 



[0135] FIG. 28 is a block diagram conceptually showing the basic configuration of a monitoring device for vehicles 
of Embodiment 3 of the present invention. In FIG. 28, the same components as those in FIG. 1 are denoted by the 
50 same reference numerals, and the description thereof is omitted here. This embodiment is different from Embodiment 
1 in that an optical flow detection section 12A detects an optical flow using a background flow estimated by the back- 
ground flow estimation section 15. This contributes to reducing the computation time of an optical flow and improving 
the detection accuracy. 

[0136] An optical flow Vi, which is the motion of an object on a camera image, is expressed by the sum of motion 
55 Vb representing the actual motion of the object and relative motion Vc caused by the motion of the camera itself. 



VI = Vb + Vc 



17 



RNSOOCID:<EP 1361543A2J_> 



aiS PAGE BLANK (ustTO) 



EP 1 361 543 A2 

[0137] If the object is immobile, that is, the object is a background, Vb is 0 and Vc is equal to a background flow. If 
the object is a moving object. Vb depends on the motion vector of the object, and Vc is nearly equal to a background 
flow. From this, It is found that when the travel amount of an object is not so large, the optical flow for this object exists 
in the vicinity of the background flow. Therefore, during the determination of the optical flow, the search can b-; nited 

5 to the vicinity of the background flow. This can narrow the search range and thus reduce the computation time 

[01 38] The method of this embodiment can also improve the accuracy of detection of an optical flow. This is especially 
effective when hierarchical Images are used for optical flow detection. Hierarchical images are used for reduction in 
computation time as described before. However, since the resolution of the image becomes lower as the hierarchical 
level is higher, the possibility that an error may arise in template matching is high. If an error arises in a high hierarchical 

10 level and erroneous detection occurs, the error fails to be absorbed in a lower hierarchical level, resulting in detection 
of an optical flow different from the actual one. 

[0139] FIG. 29 is a view diagrammatically showing the results of block matching In hterarchteal images. In FIG. 29, 
image 1 was obtained by scaling down image 0, which is an original image, and executing LPF, and image 2 was 
obtained by further scaling down the image 1 and executing LPF. Respective rectangles in the images represent blocks 
15 subjected to matching. The figure in each rectangle indicates a value of differential evaluation function between a block 
G in question and a template block R That is, assuming that the size of the blocks is mxn, the brightness of each pixel 
In the template block F is f (i, j), and the brightness of each pixel in the block G In question is g (x, y, i, j), a differential 
evaluation function E (x, y] Is represented by expression (10) or (11). 

20 . * 

E(x,y)=|:2:fe(x.y.i,j)-f(ij) (lO) 

i=l M 

25 

E(x,y)=i;Z(g(^»y'i^j)-^(i'j))' (11) 

30 That is, a block having the minimum value of differential evaluation function E (x, y) is recognized as the block corre- 
sponding to the template block R and corresponds to the optical flow in the image. 

[0140] The block matching is first performed for the image 2 having the lowest resolution. Assume that the template 
block F is currently moving upper rightward on the image. Theoretically, matching with block G (1 , -1 ) should be highest 
and the differential evaluation function E (1 , -1 ) should be minimum. In this case, however, consider that block E (-1 , 

35 1) happens to be minimum as shown in FIG. 29 due to reduction in resolution caused by hierarchical arrangement, 
texture, aperture and the like. In this event, a block corresponding to block G (-1 , 1 ) in the image 2 is set as the research 
region in the image 1 . Further, a block corresponding to a block having the minimum differential evaluation function 
value in the research region is set as the research region In the image 0. However, since no block corresponding to a 
correct optical flow exists in the resultant research region, an erroneous optical flow will be detected. 

40 [01 41 ] In this embodiment, the problem described above is overcome by adding a block In the vicinity of a background 
flow to the research region during the block matching forthe original image 0 having the highest resolution. As described 
above, an optical flow will exist in the vicinity of a background flow with high possibility. In addition, in the approaching 
object flow detection, the difference between an optical flow and a background flow is important. To state differently, 
an object in which no optical flow exists in the vicinity of a background flow is a moving object. Thus, whether or not 

45 an optical flow is a background can be recognized by including the vicinity of a background flow In the research region. 
For example, when a block having the minimum differential evaluation function value exists in the vicinity of a back- 
ground flow, the optical flow related to this block is a background. On the contrary, when a block having a differential 
evaluation function value smaller than that in the vicinity of a background flow exists in a region other than the vicinity 
of the background flow, the optical flow related to this block can be recognized as a moving object. 

50 

Embodiment 4 

[0142] FIG. 30 Is a block diagram conceptually showing the basic configuration of a monitoring device for vehicles 
of Embodiment 4 of the present invention. In FIG. 30. the same components as those in FIG. 1 are denoted by the 
55 same reference numerals, and the description thereof is omitted here. This embodiment is different from Embodiment 
1 in that the background flow estimation section 15 is omitted and an approaching object detection section 16A detects 
an approaching object by estimating spatial motion of an object. In the approaching object detection section 16A, a 
three-dimensional motion estimation portion 16c estimates spatial motion of an object, not two-dimensional motion as 
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an optical flow, using the optical flow Vi actually determined from a camera image by the optical flow detection section 
12, the motion vector T of the user's vehicle determined by the user's vehicle motion estimation section 13 and the 
space model estimated by the space model estimation section 14. 

[0143] Referring to FIGS. 31 and 32. processing by the three-dimensional motion estimation portion 16c will be 
5 described. FIG. 31 is a view diagrammaticalty showing the relationship between a vehicle 1 and a space model at time 
t-1 , in which space models MS, MW1 and MW2 as shown in FIG. 17 are presumed. Note that these space models are 
expressed in the real world three-dimensional coordinate system. A camera image taken with a camera 2 is also shown 
in FIG. 31. As described before, a point Ri in the real world three-dimensional coordinate system corresponding to an 
arbitrary point Gi on the camera image can be determined by projection using the perspective projection transformation 
10 expression (expressions (1 ) and (2)) and the space models MS, MW1 and MW2. T denotes the motion vector of the 
vehicle 1 from time t-1 until time t determined by the user's vehicle motion estimation section 13. 
[0144] FIG. 32 is a view diagrammatically showing the relationship between the vehicle 1 and the space model at 
time t. In general, the space model changes with time. Assume that the optical flow detection section 12 finds that the 
point Gi at time t-1 corresponds to a point NextCi at time t. As in the case of the point Ri corresponding to the point Gi 
15 described above, a point islextRI in the real world three-dimensional coordinate system corresponding to the point 
NextCi at time t can also be determined by projection. Therefore, a vector Vri indicating the motion of the point Ri until 
time t can be determined by the following expression. 

Voi = NextRi - Ri 

20 

[0145] The motion of the vehicle 1 from time t-1 until time t has been determined as the vector T. Therefore, by 
determining a vector (Vri - T), it is possible to determine the motion of the point Ci on the camera image in the real 
world three-dimensional coordinate system, that is, a space flow. By perfonr^ing this processing for all points on the 
25 camera image, the respective motions of all the points on the camera image in the real world three-dimensional coor- 
dinate system can be determined. 

[0146] Naturally, the space model may be obtained with various sensors and via communication as in Embodiment 
1 , or other space models may be used. 

[01 47] Based on the motion of each point on the camera Image in the real world three-dimensional coordinate system, 
30 that is, a space flow, the three-dimensional motion estimation portion 16c determines whether or not the point is ap- 
proaching the user's vehicle. Specifically, it is detemriined that the point Ci is an approaching object flow when the 
vector (Vri - T) points toward the user's vehicle, and otherwise detemnined that the point Ci is not an approaching object 
flow. 

[0148] Thereafter, the noise reduction portion 16e perfomris the same processing as that perfomried by the noise 

35 reduction portion 16b in Embodiment 1 , to detect an approaching object. 

[0149] In the above description, it was assumed that the space model was expressed in the real world three-dimen- 
sional coordinate system. Alternatively, the space model may be expressed in the camera coordinate system. In this 
case, the points Ci and NextCi on the camera image respectively correspond to points Ri and NextRi in the camera 
coordinate system. The origin of the camera coordinate system has moved by an amount corresponding to the motion 

40 vector T of the vehicle 1 during the time period from time t-1 until time t Therefore, the motion Vri of the point Ci on 
the camera image can be detemnined by the following expression. 

Vpj = NextRi - Ri - T 

45 

[0150] It is detennined that the point Ci is an approaching object flow when the vector Vpj points toward the origin, 
and othenwise determined that the point Ci is not an approaching object flow. By performing this processing for all 
points on the camera image and subjecting the results to noise reduction by the noise reduction portion 16e, an ap- 
proaching object can be detected. 

so 

<Use In combination with obstacle sensor> 

[0151] According to the present invention, an approaching object Is detected by use of image infonnation. This en- 
ables complicate detection such as whether or not an approaching object is coming closer or going away, unlike the 
55 detection with an obstacle sensor using laser, infrared, millimeter wave and the like. However, if an obstacle is near 
the user's vehicle, it is important to just detect the simple infonnation of whether or not an obstacle exists precisely 
and promptly, rather than the complicate infonnation mentioned above. 

[01 52] In view of the above, the obstacle detection may be perfomned with an obstacle sensor for a region near the 
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contact With an obstacle is most flkely to oLurThToitade '^^^^^^^^ "T!^ ' "^'^^ ^'^''^ent of 

may be incoTJorated in the bumpers or the vehicte IteSf ^ """""'^ °" orunderthe bumpers or 

fl'^ipe^op:^;:^^^^^^^^ - -'"^Therefore, when rainfa.i is recognized 

according to the present invention. This way of di^ton S^^^^^ ^'''''^''^ t° t^^^ method 

[0155] Altematively.aregionthathasbeeLlteSSasincruZ^^^^^^ 
invemion may be detected again With Obstacle seni^^^^^^^^^ 

an approaching object and prevent generation o^a faSe allnlj 'mprovethe detection accuracy for 

Eiornrhfobrj:^^^^^^^^^^^^ 

way Of detection can improve the processing speS ^ cletenn.ned by the method of the present invention. This 

[0158] As described above, the monitorinq dTv^ce oTvLiniTr h T ^ ^ '^"'"P"'^^ P^^S^^-^- 

according to the present invention can Zntat he rectio^ac^L'l'J'"' h" ^^^'^^^'"^^ using optical flows 

frLrrdi^ttr^aV^^^^^^^^^^^ 

than that specifically set out and descried ab^Tt^oZZTt ^, . ""^^ ^"^"""^ "^^"^ embodiments other 
mod.ications of the Invention which fai. .t^uCTp^r ^^^^^^^^^ '^'^'^ — all 

Claims 

1. A monitoring device comprising: 

,« ,„ <,«^„,„g „^ ,„„ ,^ ^ ^^^^^^ ^ ^^^^^ 
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if the magnitude is larger than the predetemnined value, the comparison is made with use of an angle differ- 
ence, or othenAfise the comparison is made without use of an angle difference. 

9. The monitoring device of Claim 8, wherein the predetermined value is set according to the magnitude of the back- 
5 ground flow at a corresponding position on the image. 

10. The monitoring device of Claim 1 , wherein the third means performs the steps of: 

specifying approaching object candidate flows among optical flows by comparing the optical flows with back- 
to ground flows; 

generating an approaching object candidate region by associating neighboring ones of the approaching object 
candidate flows with each other; and 

determining the approaching object candidate flows related to the approaching object candidate region as 
noise when the area of the approaching object candidate region is smaller than a predetemiined value. 

15 

11. A monitoring device comprising: 

first means for determining an optical flow from an image taken with a camera capturing the sun'oundlngs of 
a vehicle; 

20 second means for determining a space flow, which is motion of a point on the image in a real world coordinate 

system, based on the optical flow, motion of the vehicle and a space model modeling a space the camera is 
capturing; and 

third means for detecting motion of an object existing in the surroundings of the vehicle based on the space flow. 

25 12. A monitoring method comprising the steps of: 

determining an optical flow from an image taken with a camera capturing the surroundings of a vehicle; 
determining a background flow, which is an optical flow for the image under the assumption that the image Is 
a background, based on motion of the vehicle; and 
30 detecting motion of an object existing in the surroundings of the vehicle by comparing the optical flow with the 

background flow. 

13. The monitoring method of Claim 12, further comprising the step of estimating motion of the vehicle using a vehicle 
speed sensor and a steering angle sensor provided for the vehicle. 

35 

14. A program for monitoring, allowing a computer to execute the steps of: 

deterrnining an optical flow from an image ta ken with a camera capturing the surroundings of a vehicle; 

determining a background flow, which is an optical flow for the image under the assumption that the image is 
40 a background, based on motion of the vehicle; and 

detecting motion of an object existing in the surroundings of the vehicle by comparing the optical flow with the 
background flow. 

45 
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